feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] by robinnsc · Pull Request #395 · ExtendDB/extenddb

robinnsc · 2026-10-13T03:25:22Z

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #628 (part of #624).

Stacked on #394 (base branch feat/backup-format-streaming): SnapshotInfo is #394's SnapshotDescriptor and the orderability test is its KeyOrder::resolve. The diff shows this PR's single commit on top of #394; GitHub retargets it to main when #394 merges.

What

The one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code in #384 keeps its own copy of this logic; the engine-side backup job (next PR) is the first caller.

Trait (crates/storage/src/snapshot.rs):

pub struct TableSnapshot {
    pub description: TableDescription,   // the storage-level description at the snapshot instant (not the API response: restore_summary may be set, table_throughput_mode_summary is not, and the counts are the stats pass's)
    pub info: SnapshotInfo,              // = extenddb_core::backup_format::SnapshotDescriptor { kind, marker }
    pub key_ordered: bool,               // items arrive in base-table key order
    pub items: BoxStream<'static, Result<Item, StorageError>>,  // owns the transaction/session
}
pub trait SnapshotEngine: Send + Sync {
    fn open_table_snapshot(&self, account_id: &str, table_name: &str)
        -> BoxFuture<'_, Result<TableSnapshot, StorageError>>;
}

Dropping the stream releases the backend transaction or session. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. key_schema_is_orderable is KeyOrder::resolve(..).is_ok(), so a stream a backend marks ordered is one the manifest parser accepts, and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. The trait doc lists each backend's errors (Cassandra answers Unsupported before it resolves the table). StorageError::SnapshotExpired is new, for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient.

Indexes. The description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since #401 UpdateTable creates the data table first, so this is a guard). A restore recreates the indexes in the description and builds them from the items, so a CREATING index comes back complete; an index left out would not come back at all.

PostgreSQL. On one shared database the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. With a separate data database the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both layouts. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, and an open over the bound is LimitExceeded at once, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key; a fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit), and the byte budget is the backup job PR's. A table created before #361 declared the key columns COLLATE "C" keeps the database collation; ordering it bytewise would sort the whole table before the first row while the lock, connection and permit are held, so such a table is streamed unordered with a warning. Only a collation named "C" or "POSIX" counts as bytewise; "default" on a C database streams unordered, which is complete. SQLSTATE 40001 (a recovery conflict on a standby) and 72000 (snapshot_too_old on a primary, PostgreSQL 16 and earlier) map to SnapshotExpired. The stream holds ACCESS SHARE until its last fetch, so a DROP TABLE waits behind it; DeleteTable takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that). On both layouts the synchronous path takes every drop's lock before the catalog removal commits. It deletes the table's pending queue rows first, waiting for in-flight claims with no timeout, as #401 intends. It then drops the base, GSI, LSI and vector-index data tables, base first, within one 1 s budget for the group (SYNC_DROP_LOCK_TIMEOUT; before each DROP, lock_timeout is the time left). On the shared layout that runs in the catalog transaction behind a savepoint. With a separate data database it runs in a data transaction opened first, and a task the request awaits commits the catalog and then that transaction, so a cancelled caller cannot stop between the two. A lock wait that fails (55P03, a deadlock victim 40P01, or a statement_timeout's 57014) rolls the drops back and takes the asynchronous path with the row still held. A write transaction, a SHARE holder, an index build or a reader that starts after the probe therefore costs the request at most the 1 s, and no data table is left without its row; the one remaining window is a process crash between the two commits on the separate layout, one round trip wide. The base table goes first because a write holds it before it writes an LSI or a synchronous GSI: with the index tables first, a write in progress waited about 1 s on the drop, and the drop went asynchronous. On the shared layout a DeleteTable probing between an open's snapshot and its LOCK does not see the open; if the LOCK is granted first, the drop times out after 1 s, is rolled back with the removal, and DeleteTable returns DELETING (in 80 unaided open-and-delete pairs, the delete first in half of them, 0 left a data table without a row on either layout; 39 to 40 deletes ran synchronously and 40 to 41 asynchronously). The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred, also after a restart: startup recovery runs the same pass and wakes the poller when it deferred a drop. A drop is deferred when the probe finds another session holding any lock on the table's relations, or when the drop fails in a lock wait (55P03, 40P01 or 57014); any other failure is logged and left to the normal window and the 60 s idle pass. Each pass probes a held table and skips it without waiting, and drops a free one within the same 1 s budget, so every held table is retried about once a second however many are held, and a CreateTable that is due is activated on the same cadence (1.1 s with four drops deferred). A drop that DeleteTable deferred is due at once, not after control_plane_delay_seconds. The active loop waits on the wake as well as on its timer, so a snapshot's or an in-catalog backup's release has the drop retried at once, and any holder's release is followed within about a second (the maintainer measured 51.8 s before for a snapshot stream, and round 6 57 to 60 s for the in-catalog backup and a plain reader; now 108 to 166 ms after a snapshot or backup release or a plain reader's, 0.94 to 1.08 s after a release that spans a restart (the reader is released only once the poller is past its window), and 108 to 164 ms for the first of four held tables, in the tests). The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the default (0.25 on every backend), the bound and the retry cadence, the same-name CreateTable refusal until then, and a two-step query, over every relation of the table, that finds any session holding a lock on them and its lock mode. A DeleteTable of a table that is already DELETING returns its description and changes nothing.

SQLite. One deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened. A TRUNCATE checkpoint meanwhile waits out the busy timeout and reports busy = 1; the module doc gives the measured reply. build_table_description_on reads the definition, including #384's restore_summary, on that transaction. The stream holds a slot from #384's backup_server_slots until its read transaction ends (the short page that shows the end, or the first error) or it is dropped (the pool arithmetic that keeps half the pool free counts snapshots); with none free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them; the pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index; a page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it has yielded its last item or is dropped, so the slots bound how many decoded copies exist. This is the shape #384's create_backup_single_connection uses, so CreateBackup keeps working on sqlite-memory. SQLite DeleteTable of a DELETING table (control_plane_delay_seconds of 1 or more) returns its description too.

MongoDB. A session started with snapshot: true reads the tables document, the indexes documents and the base collection at one cluster time, which is the marker. Every read sets readConcern: snapshot explicitly, because the driver only fills it in when neither the operation nor its database carries one, and the production catalog handle carries readConcern: local (as any handle does under a URI with readConcernLevel=); on that handle a read without the level fails: the first reports no snapshot time, which the open refuses naming the read concern, and later ones are refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported; that refusal and the no-snapshot-time refusal each name their own cause. The index read in the session maps 239/246 like the others. Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes; CursorNotFound maps to Internal). key_ordered is false: a composite key's _id is a netstring of the key text, not the format's order.

Cassandra. open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog path stays its backup path until the design settles it (#396 section 3).

Test hooks. A fetch-size knob, a round-trip counter and a pause at two points of the open (AfterDefinition, BeforeItems) live in extenddb_storage::snapshot::paging; every backend reads the fetch size there and calls the pause. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with extenddb-storage's test-hooks feature, which each backend crate's tests enable through a dev-dependency on itself (on MongoDB through a new snapshot-test-hooks, so the crate's existing coordination test-hooks stays off in its unit tests, as on main); production builds carry none of them. PostgreSQL also gains test-hooks entries spawn_control_plane_poller_for_tests and spawn_control_plane_poller_with_active_poll_for_tests (the production poller, the second with its active retry interval as a parameter), recover_control_plane_at_startup_for_tests, control_plane_drop_attempts_for_tests (a per-engine count of drop attempts, a held table the probe skipped counted) and arm_delete_table_pause (holds DeleteTable between its probe and its drops), and re-exports, under test-hooks, the data relation names and apply_all_migrations (the three extenddb migrate runners), so tests/table_snapshot.rs binds the crate's lists instead of copying them. The two pauses (the open's and DeleteTable's) are one generic type, extenddb_storage::test_pause::Pause, and snapshot_conformance::PageSize is the one page-size guard the three backends' tests use. tests/table_snapshot.rs fails instead of skipping without a connection string when EXTENDDB_TEST_PG_REQUIRED is set, which integration.yml sets. The trait doc says why SnapshotEngine has no default body and StorageError is not #[non_exhaustive]: both breaks are meant. devtools/run-mongodb-tests publishes its container on 127.0.0.1 only.

Why

#396 section 3. An earlier draft of the trait returned items only and left the definition to a separate DescribeTable; a GSI added or the table deleted between the two calls produces a manifest that does not match its data files. Returning both from one operation is the only way the backup job can promise the manifest describes the data, and it has to be a backend obligation because the mechanism differs per backend.

Four adversarial review rounds on this PR found, and this revision fixes: the conformance check was inert at the production page size, then depended on the backend test setting the page size, then could not observe whether the knob had taken effect (it now sets the shared knob and asserts from the counter that a second fetch happened); the MongoDB engine inherited the client's read concern and the constant for SnapshotTooOld was SnapshotUnavailable's; an ACTIVE-only index filter dropped a GSI mid-backfill; the PostgreSQL ORDER BY lacked COLLATE "C", so a pre-#361 table streamed in locale order under a key_ordered flag and the design's restore would have refused a valid backup; the PostgreSQL open took two catalog connections and could deadlock a full pool; the MongoDB tests dropped fixed database names, so a URI in the environment could destroy another instance's data; the SQLite path held the writer lock across a pool acquire and bypassed the slot accounting; a missing sort-key attribute definition made the stream page by hash key alone and lose rows; SnapshotInfo and the orderability rule were second copies of #394's definitions. The fourth round: the MongoDB counter echoed the batch size the stream asked for (a cursor without batch_size passed); a synchronous DeleteTable during a stream could orphan the data table; a half-created GSI left an unused attribute definition CreateTable refuses; a shared catalog/data pool could deadlock concurrent opens; the Cassandra test created no table; the harness's key-order oracle was a third statement of the order. An earlier review by the maintainer then found: nothing checked that the definition and the items come from one instant or that the stream read after the writer committed (READ COMMITTED, a missing FOR SHARE, a MongoDB read outside the session, and a SQLite stream that read everything during the open all passed); only one key shape was tested; with a separate data database the bound was the whole pool, and ten snapshots made GetItem fail after 30 s; DeleteTable probed only the base data table, so a reader on a GSI table left the catalog row gone and the drop blocked; the collation clause turned a legacy table's read into a full sort; a MongoDB decode error did not end the stream and a standalone server reached the client as a 500; the SQLite refusal blamed backups when snapshots held the slots; and sqlite-memory lost a working CreateBackup. The fifth round found: on the shared layout DeleteTable took a second connection from the pool it already held one from, and twenty concurrent deletes on a ten-connection pool timed out; the open took two connections from one pool on the shared layout; four release paths (a drained stream, a fetch error, an open that failed or was cancelled) did not wake the poller; the one-instant scenario passed with the pauses outside the read window; the SQLite rowid page was untested and the MongoDB counter counted the GSI backfill's reads; a foreign test's open could take the armed pause; N range keys were not among the key shapes; in-memory SQLite copies were not bounded by the slots. Round 6 found that the one-second DeleteTable bound held only for readers that woke the poller (a snapshot stream), while a plain read, pg_dump or the in-catalog backup left the row DELETING until the 60 s idle pass; the poller now stays active while a drop is deferred, the backup wakes it too, and the conformance check asserts the branch it depends on, compares the description with DescribeTable, and bounds every drain. The maintainer's review of 2026-10-28 found: on one shared database an open's window between its snapshot and its LOCK let a synchronous DeleteTable commit the removal and wait in DROP TABLE with no timeout (6 of 80 unaided races left the data tables without a row); the release wake was inert because the active poller slept on its timer alone; startup recovery discarded the deferred count, so a reader spanning a restart left the row DELETING until the 60 s idle pass; every drop failure kept the poller in its one-second loop; the troubleshooting query matched only the base table; and the conformance check cleared restore_summary. Round 7 found: with a separate data database a DeleteTable that met a write transaction, a SHARE holder, an index build or a queue claim waited with no bound after the catalog commit, and a caller cancelled or timed out in that wait left the data tables without a row; a lock timeout per DROP let a table with n indexes stall its base table n seconds; the poller retried deferred drops one after another at 3 s each, delaying every other transition by 3 s per held table; the troubleshooting query found only readers; DeleteTable of a DELETING table returned ResourceNotFoundException; a deferred drop waited out control_plane_delay_seconds; a deadlock victim or a statement timeout in the drop was an InternalServerError; and SQLite kept a finished or failed stream's backup slot.

Testing done

extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared fetch size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open, then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. It asserts 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, that the counter advanced after the writer committed, key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare when ordered, and that a second snapshot after the writer stops shows the final round. Then, once per pause point and on a fresh table, it holds the open while it commits a write W1, an UpdateTable adding a GSI and a write W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts, the snapshot's description must equal the storage layer's describe_table with the running counts and table_throughput_mode_summary cleared; restore_summary is compared, and each backend's suite also compares a table given a restore record (assert_snapshot_describes_the_table). Every drain is under a 120 s deadline naming the stage and the items read, and a test waiting for the hooks lock gives up when one holder has kept it 600 s, naming the wait. Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3, with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. These mutants fail it: a fresh transaction per page or every page read during the open on SQLite, and >= in its hash-only or binary range keyset; OFFSET/LIMIT in READ COMMITTED, READ COMMITTED with the cursor, and no FOR SHARE on PostgreSQL; the data cursor or the index read outside the session on MongoDB; the pauses moved outside the read window on SQLite or MongoDB; the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion); a description built from a second code path (the DescribeTable comparison); a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. 27 hand mutants were built for round 4; 25 fail on the assertion they target, and the two that remove COLLATE "C" from the PostgreSQL clause pass because, with legacy tables streamed unordered, the clause is equivalent on every table that is ordered (kept as defence in depth and marked so). Round 5: 18 more mutants (DeleteTable's probe on a second pooled connection, the barrier shape on the shared layout, each wake site removed, the 42P01 mapping, the fetch-error path keeping its slot, no LOCK TABLE, the last page keeping its transaction, the barrier released before the pause, the pauses moved outside the read window on SQLite and MongoDB, rowid >=, the in-memory copy dropping its slot, the MongoDB counter counting every find); 17 fail on the assertion they target, and the barrier released between the pause and the pin passes, because no statement sits between them and UpdateTable takes over a second after the barrier goes. Round 6: the poller breaking at its deadline regardless fails the plain-reader test on both layouts (two earlier shapes of that test, an 8 s and a 12 s hold with the poller started before the DeleteTable, did not catch it: the window's last drop attempt was still waiting out its lock timeout, and the DeleteTable's own wake chained a second window); the SQLite pauses moved before the transaction with a 2.5 s UpdateTable fail the branch assertion; a SQLite open blanking the ARN, id and summaries fails the description comparison; a SQLite keyset stream without its end branch fails the drain deadline. For the maintainer's review of 2026-10-28: 9 mutants (the shared-layout drops after the commit; the notify removed from release and from finish; the active loop ignoring the notify; startup not waking the poller; every drop failure counted; the snapshot dropping restore_summary on PostgreSQL, SQLite and MongoDB), each failing on the assertion it targets. Round 7: 17 mutants (the separate layout's commit-then-drop order; a lock timeout on the pending-row delete; a timeout per DROP; index tables dropped first; the poller's probe skipped; activation after the drops; the DELETING filter on PostgreSQL and on SQLite; the deferred row scheduled after the delay; 40P01 or 57014 left out of the classifier; the poller idle at its window's end; DeleteTable's probe never finding a reader; the SQLite slot kept on the short page, on the error and in the in-memory copy; description_diff keeping commas). 16 fail on the assertion they target. Activation after the drops passes, since with the probe the drops no longer take time.

Passes locally on PostgreSQL 16 (both pool layouts), SQLite (the shared check on a file-backed database; the in-memory copy has its own tests, since a stream that reads every page during the open fails the check's fetch-after-commit assertion by design), a MongoDB 7 replica set (devtools/run-mongodb-tests --mongo-only) and a standalone mongod, and Cassandra 4.1:

PostgreSQL: crates/storage-postgres/tests/table_snapshot.rs, 78 tests (and two ignored race measurements): 36 run on both layouts (on_both_layouts!): consistency; marker and key order; legacy-collation tables streaming unordered and complete on a scratch database checked not to be C-collated, beside an ordered one; DeleteTable with only a GSI or a vector-index data table locked taking the asynchronous path; an unparseable row ending the stream and freeing its permit; a GSI without its data table left out with its attribute; DeleteTable during a snapshot returning DELETING; twenty concurrent DeleteTables on a ten-connection pool; an open held while nine DeleteTables queue; an open completing with nine of ten connections held; the production poller at a 10 s active poll woken during its pause, and the table gone within 3 s, by a dropped stream, a drained stream, a fetch error, an open cancelled at BeforeItems and the in-catalog backup; a reader across a restart; a drop failing on a dependent view kept to the normal window; a restored table's restore_summary; a plain reader holding ACCESS SHARE for 10 s past the poller's window, sending no wake, with the table gone within 3 s of its release; a fetch error freeing the permit; DeleteTable at BeforeItems taking the asynchronous path; a drained stream out of its transaction before its end is polled; a ROW EXCLUSIVE and a SHARE holder sending DeleteTable down the asynchronous path within about 1 s with no data table left without its row; a DeleteTable cancelled during that wait leaving the table ACTIVE with its data; write holders on the base and three GSI tables released 0.8 s apart taking it there at about 1 s; a write holding the base and then writing the GSI table finishing first, with the table removed in the request; a claimed queue row holding DeleteTable until it ends; a probed reader sending it asynchronous in milliseconds; a statement timeout and a deadlock victim taking the asynchronous path; DeleteTable of a DELETING table returning its description and changing nothing; a deferred drop due at once; four held tables probed with no lock wait, the first released gone within 2 s and a CreateTable ACTIVE within 2 s. Plus five held snapshots on a ten-connection pool on each layout, with GetItem and DescribeTable unblocked, the sixth LimitExceeded and an open succeeding after one is dropped; the SQLSTATE mapping raised on the live server; and, on the shared layout, a snapshot open or a GSI-only reader locking after DeleteTable's probe sending it down the asynchronous path, and a table dropped before the open's LOCK being TableNotFound. The two ignored race measurements delay the delete or the open in alternate pairs and assert both outcomes occurred. Every DeleteTable runs under a caller timeout (300 ms in the cancellation test). Added to the direct-storage step in integration.yml. Scratch databases are dropped by a Drop guard on success or panic. The full storage-postgres suite (256 tests), fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401's GSI build tests included, passes.
SQLite: 11 in-module tests (consistency; key order; the in-memory copy releasing its connection; in-memory copies holding their backup slot; a damaged attribute definition streaming unordered and complete at a page of seven with no duplicates and at least six fetches; slots shared with CreateBackup and its refusal text; the WAL checkpoint reply while a snapshot is held; a restored table compared on a file-backed and an in-memory engine; a drained stream and a stream that hit an error freeing their slot while still held; an in-memory copy freeing its slot with its last item), run by cargo test --workspace; scratch files under the test binary's target directory, removed by a Drop guard.
MongoDB: 11 in-module tests: live tests on per-test scratch databases (snap<uuid>_cat/_data, built the way fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's live tests build theirs with the production catalog options, with the GSI backfill worker running and the command monitor feeding the counter, dropped by a guard on success or panic), gated on EXTENDDB_TEST_MONGODB_URI and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, holding the snapshot hooks lock (consistency; a restored table's restore fields compared; readConcernLevel=majority in the URI; an undecodable document ending the stream; reads without the snapshot level failing with code 72, with the error text checked; the round-trip counter counting only the snapshot stream, so a GSI backfill leaves it unchanged; a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod it starts, and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it); and unit tests pinning the error-code mapping from reply documents. A new integration-mongodb.yml job (in the fan-in) starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod, both on 127.0.0.1 only, and runs them.
Cassandra: a direct test asserting Unsupported naming the backend on an existing table and a missing one.
Storage: unit tests for drop_deleting_indexes (GSIs and vector indexes).
Engine: SnapshotExpired maps to ServiceUnavailable (storage_error_mapping_tests).
Wire: tests/test_table_operations.py's DeleteTable of a DELETING table returning its description, against PostgreSQL and SQLite servers (the state written into the catalog); it skips on MongoDB and Cassandra, which never leave a table DELETING.

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy ... -- -W clippy::pedantic                                             # 0 findings on the lines this PR adds and in the files it adds (storage crates and extenddb-engine)
cargo check --workspace --all-targets                                                # with test-hooks via dev-deps
cargo check -p extenddb-storage -p extenddb-storage-postgres -p extenddb-storage-sqlite -p extenddb-storage-mongodb -p extenddb-storage-cassandra --lib   # without
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo test -p extenddb-storage -p extenddb-engine -p extenddb-server                 # 132, 179, 202 passed
cargo test -p extenddb-storage-sqlite --lib snapshot_engine                          # 11 passed
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres --test table_snapshot   # 78 passed, 2 ignored
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres                         # 256 passed
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 EXTENDDB_TEST_MONGODB_STANDALONE_URI=... cargo test -p extenddb-storage-mongodb --lib snapshot_engine   # 11 passed
EXTENDDB_TEST_CASSANDRA=required cargo test -p extenddb-storage-cassandra --test direct_integration snapshot_engine   # 1 passed

Not changed here

The design's max_concurrent_jobs is the caller-side bound, in the catalog PR; this PR's per-server bounds (SQLite's slot, PostgreSQL's permit) only keep snapshots from exhausting a pool.
The in-catalog backup path in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 does not call this trait.
The PostgreSQL and SQLite fetches are bounded by rows, not bytes; the byte budget is the backup job PR's (design section 7).
With a separate data database a process that dies between the catalog commit and the data commit of a synchronous DeleteTable leaves the data tables without a row; the window is one round trip.
The drops take the base table before the index tables, not after: a write holds the base before it writes an LSI or a synchronous GSI, and the other order made such a write wait about 1 s on the drop.
Four issues round 7 found outside this diff are present on main (cf321f0) and out of scope here:
- extenddb catalog-check --fix expects data relations named _ddb_{account}_{table} and _ddb_{account}_{table}__gsi__{index}, while main names them _ddb_{table_id} and _ddb_{index_id}, so it reports every live data table as orphaned and --fix drops it; it does not see _ddb_vec_ tables either.
- On the shared layout the catalog migration's stream_shards.table_id references tables(table_id), and CreateTable inserts the shards on its own connection before its catalog transaction commits, so a CreateTable with a stream specification fails the foreign key with an InternalServerError (extenddb init does not produce that layout).
- UpdateTable that creates a GSI awaits the backfill inside the request, so on a large table the client's read timeout expires first (boto3's 60 s against 20,000 items); the build itself continues server-side.
- Query and Scan with IndexName resolve the table and then the index in separate statements, so a DeleteTable committing between them yields ValidationException ("The table does not have the specified index") instead of ResourceNotFoundException.
Attribute pruning for a left-out GSI is not shared between the snapshot and the in-catalog backup (they prune different types); backup_engine::abort_restore builds its own drop list in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's code.
cargo clippy on the local 1.98 toolchain did not fire the chunks_exact lint in storage-sqlite/src/vector_search.rs seen in an earlier round; CI runs stable and 1.88.0 (test.yml).
docs/design/04-component-storage.md (DeleteTable described as always asynchronous) and docs/differences-from-dynamodb.md (a 5 s default delay) are pre-existing inaccuracies not edited here.
The CodeQL alert on this PR (cleartext-storage-database, storage-sqlite/src/backup.rs) is in a fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 test on main that this PR does not touch.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic) for the files this PR adds
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, section 3. Storage trait: SnapshotEngine is added to the StorageEngine supertrait bound and implemented by all four in-tree backends; StorageError gains SnapshotExpired.

Breaking changes

Out-of-tree storage backends must implement SnapshotEngine and handle the new StorageError::SnapshotExpired variant in any exhaustive match.
PostgreSQL DeleteTable returns DELETING and finishes in the background, instead of synchronously, when another session holds a lock on the table's data or index tables that its drops cannot get within about 1 s, on both layouts: at once for a reader (a table snapshot, the in-catalog backup, a Scan or Query, pg_dump, a plain read), after the 1 s for an open write transaction, an index build or VACUUM, and also when the drop is a deadlock victim or meets a statement_timeout. The table is gone within about a second of that session's release (at once when a table snapshot or in-catalog backup ends; also when the server restarted in between). Until then DescribeTable reports DELETING, ListTables lists it, a CreateTable of the same name is refused with ResourceInUseException, and a DeleteTable of it returns its description, which on PostgreSQL and SQLite used to be ResourceNotFoundException. The response shape is unchanged. No wire, CLI or on-disk change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>

…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, two review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main 744ea29 after #384 merged in its final form and the Cassandra backend landed. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Reconciled with #384 as merged: the restore job record extends #384's table_restores / provenance fields rather than adding a table; the backup id (14-digit, new_backup_id) and ARN parser #384 introduced are the one implementation, moved into extenddb-core by PR 4 and re-exported by storage; the claim names #384's RESTORE_LOCK_SEED, 15 s heartbeat and slot semaphores as what it replaces; defaults follow it (15 s heartbeat, 300 s stale window). Cassandra (#338) implements SnapshotEngine as Unsupported and keeps its logical backup until a token-range snapshot is accepted or the backend's backups are documented as unavailable, decided before PR 14. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), three review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main 744ea29 after #384 merged in its final form and the Cassandra backend landed. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE- only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time (stricter than the service's two- minute window). The backup id is minted from the instant the snapshot pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets only); no row ever exists unowned (the MongoDB row is born through an upsert pipeline so its first heartbeat is the server's clock). Heartbeats are a timer, staleness is measured on the database's clock, and any heartbeat that does not confirm the claim aborts the job. A job runs the failure path's delete_prefix only after its FAILED write matched a row; otherwise the claim holder owns the prefix. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING, leaves heartbeat_at untouched, and deletes the prefix only after the stale window; its duties are listed once, in section 4. A sink error fails the backup; there is no per-file restart, which the one-shot stream and the memory bound rule out. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically in a detached task; TableAlreadyExistsException or TableInUseException on a name race, as #384 already returns; LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional and precedes the deletion of the PostgreSQL restore_progress rows, which live in the data database and cannot share its transaction. DeleteTable on a restore target fails the job (a change from #384, which refuses it); multi-part-key tables are restored (#384 refuses them). The override paragraph now states what #384 ships: every override but VectorIndexOverride is applied, and the new path keeps that. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; #395's PostgreSQL ORDER BY carries COLLATE "C" so tables created before #361 stream in the format's order. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated; list and delete_prefix resolve their prefix with the same no-follow walk as a key and tolerate entries a concurrent deleter removed. The S3 conformance suite is mandatory in CI against moto, with the 301/403/abort paths driven through an interceptor. Store settings live under [backup.store], matching #393. - ListBackups keeps #384's handler (full request already on main) over the catalog index merged with the legacy reader; sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs, a memory formula in bytes with the item-size term, and a pool-size rule that clamps and warns are specified. Restore and Describe never read the store without a catalog row; backup sync is the path in. - Reconciled with #384 as merged: the restore job record extends #384's table_restores / provenance fields rather than adding a table; the backup ARN parser and formatter #384 introduced are the one implementation, moved into extenddb-core by PR 4 and re-exported by storage (new_backup_id stays in storage and calls the shared formatter); the claim names #384's RESTORE_LOCK_SEED, lock key, 15 s heartbeat and slot semaphores as what it keeps or replaces; the 300 s stale window is new (#384 uses 60 s). Cassandra (#338) implements SnapshotEngine as Unsupported and keeps its logical backup until a token-range snapshot is accepted or the backend's backups are documented as unavailable, decided before PR 14. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend. Also adds the missing rows for designs 12 and 13 to docs/design/README.md. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), four review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. PostgreSQL bounds open snapshots per server and refuses more with LimitExceeded; DeleteTable takes its asynchronous path while a reader holds the data table. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant it pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets and stale spools. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store, never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed; #395's PostgreSQL ORDER BY carries COLLATE "C". - Store (#393): owner-only root and directories re-checked by every operation, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code, settings under [backup.store]. - Configuration: the default store root is per backend (beside the database file on SQLite); PostgreSQL, MongoDB and Cassandra refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy reader until PR 14. The memory formula counts the fetch buffer and caps a restore batch at one file; purge holds every job permit. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan

I ran every test the description lists on PostgreSQL 16, SQLite, a MongoDB 7 replica set and Cassandra 4.1 (8, 5, 5 and 1 pass), built mutants of each backend's snapshot code and of the shared check, and compared head and main PostgreSQL servers for DeleteTable under concurrent readers. The SQLite snapshot excluded 900 writes and a DeleteTable committed after the open. Descriptions matched DescribeTable member by member on SQLite and MongoDB, N and B pages came back in KeyOrder::compare order, and MongoDB's expiry mapped to SnapshotExpired on a live server. The base-table DeleteTable change closes an orphan that main has. Nothing in the tree opens a snapshot yet, so no request reaches the items below today.

Eight items are inline. Smaller items:

storage-mongodb/src/snapshot_engine.rs:18 has the read-concern sentence I commented on in #396 line 157: without the explicit level the reads fail.
table_snapshot.rs:271 passes without COLLATE "C" on a server whose default collation is C. Assert the scratch database's datcollate first.
devtools/run-mongodb-tests --mongo-only exits with an unauthenticated replica set published on every interface. Publish it on 127.0.0.1, as the printed URI and the CI job already assume.
The storage-mongodb self dev-dependency turns on the crate's existing test-hooks feature, so its tests no longer build the production commit and backfill path. Give the snapshot knob its own feature.
After a 20 s reader released, the table stayed DELETING for 51.8 s, because the retry waits for the control-plane poller's idle pass.
The SQLite module doc should say that PRAGMA wal_checkpoint(TRUNCATE) reports success while a snapshot is held. The WAL grew to 8.67 MB for 900 items of 400 bytes.
MongoDB snapshot errors carry a hex dump of the BSON reply. Build the message from code_name and errmsg.
Cassandra answers Unsupported before it resolves the table, where the others answer TableNotFound. Say so in the trait's error list.

Once the inline items are fixed, I will approve.

LeeroyHannigan · 2026-10-27T03:30:51Z

+    let snapshot_description = snapshot.description;
+    let mut items = snapshot.items;
+
+    // Writer: runs only after the snapshot is open. Signals once the deletes,


No test checks that the definition and the items come from one instant, or that the stream reads after the writer commits. The writer starts after open_table_snapshot returns and never changes the catalog: at READ COMMITTED, or without FOR SHARE on the tables read, all 8 PostgreSQL tests pass, and with the data cursor or the index read outside the snapshot session, all 5 MongoDB tests pass. A SQLite stream that reads every page during the open, each in its own transaction, also passes, because the counter only shows fetches before the pause. Add a test-hooks pause between the definition read and the data read in which the check commits a write and starts an UpdateTable, and assert after the drain that paging::fetches() advanced past its value when committed_rx resolved; that assertion passes on all three backends as they are.

LeeroyHannigan · 2026-10-27T03:30:51Z

+    (pk, sk)
+}
+
+/// The table the check creates: composite key (`pk` S, `sk` N) with one GSI.


Every test table has this key shape, so the others are untested. Removing sk_s COLLATE "C" or the hash-only pk COLLATE "C" passes all 8 PostgreSQL tests, and changing SQLite's hash-only or binary range-key keyset comparison from > to >=, which repeats the last item of every page, passes all 5 SQLite tests. The head handles those shapes correctly; run the same checks on hash-only S, N and B keys and on S and B range keys at a small page size. The vector-index filter in drop_deleting_indexes is untested too: removing it passes snapshot.rs:225.

LeeroyHannigan · 2026-10-27T03:30:51Z

            restore_server_slots: Arc::new(tokio::sync::Semaphore::new(server_slots)),
            restore_server_slot_count: server_slots,
            restore_account_slots: backup_engine::RestoreAccountSlots::default(),
+            snapshot_open_slots: Arc::new(tokio::sync::Semaphore::new(


With a separate data database this bound is the whole data pool. On a ten-connection pool with ten snapshots open, get_item and delete_table each failed after 30.0 s with "pool timed out while waiting for an open connection", which reaches the client as a 500. Keep the bound below the pool in both branches. The test for this permit runs its ten opens in one join_all, and dropping the permit when open_table_snapshot returns passes all 8 tests; hold five snapshots, assert the sixth is LimitExceeded, then drop one and assert an open succeeds.

LeeroyHannigan · 2026-10-27T03:30:51Z

+        // so the probe sees every snapshot that can still hold the table. A
+        // queue worker's row-level claim (`ROW SHARE`) is not matched: the
+        // drop waits for it, as `drop_table_storage` intends.
+        let data_table_in_use: bool = sqlx::query_scalar(


This probe binds only the base data table, while the synchronous path also drops the GSI and vector-index data tables. With a session holding ACCESS SHARE on a GSI data table only, DeleteTable committed the catalog removal and then blocked in DROP TABLE with no lock timeout; after 900 s tables had no row and both data tables remained. A Query or Scan with IndexName and the GSI backfill worker hold that lock. Probe every relation the drop touches.

LeeroyHannigan · 2026-10-27T03:30:51Z

+
+/// `ORDER BY` columns for the format's key order, or `None` when the key
+/// shape has none (`key_schema_is_orderable`). The text columns carry an
+/// explicit `COLLATE "C"`: tables created since the key columns were declared


On a table whose key columns keep the database collation, this clause turns the ordered read into a Seq Scan and an external merge sort (7,224 kB on disk for 100k rows at work_mem 4MB), so the first FETCH waits for the whole sort while the lock, the connection and the permit are held. The same table without the clause is an Index Scan. Recollate or index such a table before streaming, or report it unordered, and say so at line 50.

LeeroyHannigan · 2026-10-27T03:30:51Z

+//! `CREATING` index with a data table is kept.
+//!
+//! The returned [`TableSnapshot`] owns the open transaction, session, or
+//! connection for as long as its `items` stream is alive. Dropping the stream


Dropping the stream outside the runtime panics on SQLite and PostgreSQL. With rows left, sqlx's PoolConnection drop spawns the connection's return and fails with "this functionality requires a Tokio context". A MongoDB stream dropped the same way returns in under a millisecond, so a caller tested on MongoDB panics on the sqlx backends. State this here for out-of-tree backends, or give the type a release path that does not depend on where it is dropped.

LeeroyHannigan · 2026-10-27T03:30:51Z

+            return None;
+        }
+        match state.cursor.next(&mut state.session).await {
+            Some(Ok(doc)) => Some((document_to_item(&doc), state)),


A document this decoder cannot read yields an error and the stream continues: with one such document the stream returned its 10 items, one Document missing item_data field error, then ended normally, where SQLite ends the stream at the error. Set done = true in this arm and say in the trait that the first error ends the stream. Separately, a standalone server refuses the first read with NotAReplicaSet (code 123), which reaches the engine as a 500 instead of the Unsupported sentence at line 121; map that code.

LeeroyHannigan · 2026-10-27T03:30:51Z

+        // would hold every connection and every writer would wait out the
+        // pool's acquire timeout under the engine write lock. Refuse rather
+        // than queue; the caller bounds concurrency.
+        let permit = std::sync::Arc::clone(&self.backup_server_slots)


This is the permit create_backup waits 30 s for. With the default pool_size = 10, four held snapshots made CreateBackup wait the full 30 s and answer "Too many backups are in progress on this server (at most 4 at once)" with no backup running, and with a pool_size of 4 or less one held stream blocks every CreateBackup. Name snapshots in that message and update the pool_size doc in crates/storage-sqlite/src/config.rs.

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), five review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop. On sqlite-memory the snapshot copies the table on the one connection, so CreateBackup keeps working there. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant it pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first and is only a hint). Every duty that deletes a prefix acts only on rows of the configured store. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store (backup sync --move-store re-points the rows of a copied store), never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed. The line cap is 32 MiB, sized from the measured worst case, with its decode memory stated; the compressed size is enforced while reading. - Store (#393): owner-only root and directories re-checked by every operation, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the write grants, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint needs allow_insecure_endpoint) and an IAM example with the list actions on the bucket. - Configuration: the default store root is per backend (beside the database file on SQLite); PostgreSQL, MongoDB and Cassandra refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy reader until PR 14. The memory formula counts the fetch buffer and caps a restore batch at one file; purge holds every job permit. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), five review rounds (the fifth on 2026-10-27, Lee's reviews among them) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; restores hold their own half-pool slots. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop, so its DROP does not wait behind a snapshot or backup, with two stated gaps: a short GetItem, Query or Scan between the probe and the DROP, and on the shared layout a probe between an open's snapshot and its LOCK. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries older than the stale window with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first, carries its write time and is only a hint). Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store and checks the store id in extenddb-store.json, so an unmounted volume or a recreated bucket fails loudly; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store and --prune-missing take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 states the decode memory from #394's DataFileDecoder doc: about 76 bytes per line byte for flat or nested lists and at most about 128 for any shape, about 4 GiB at the cap, after #394 stopped the deserializer copying nested values. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; localhost over http needs force_path_style) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite); PostgreSQL, MongoDB and sqlite-memory refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy create until PR 14, whether or not [backup.store] is set. The memory formula is in item text bytes, and PR 8a adds a byte budget to the PostgreSQL and SQLite snapshot streams; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10); the spool default is per backend. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section names the companion heads the copied values come from. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), six review rounds (the sixth on 2026-10-28, Lee's reviews among them) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop, so its DROP does not wait behind a snapshot or backup, with two stated gaps: a short GetItem, Query or Scan between the probe and the DROP, and on the shared layout a probe between an open's snapshot and its LOCK; the control-plane poller stays active while a drop is deferred, so the drop follows any reader's release within about a second. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries older than the stale window with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first, carries its write time and is only a hint). Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store and checks the store id in extenddb-store.json, so an unmounted volume or a recreated bucket fails loudly; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store and --prune-missing take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load writes through the backend's own item path, so the table is laid out as a natively created one). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path (three #384 tests change in PR 8a/9a). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; localhost over http needs force_path_style) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), and kind = "disabled" is the opt-out. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a adds the byte budget SNAPSHOT_FETCH_BYTES (64 MiB) to the PostgreSQL and SQLite snapshot streams; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan

I re-ran the items from my review against this head on live databases, on both PostgreSQL layouts, and found no item open. The one-instant check now fails for each reader shape I named, each on its own assertion; the SQLite >= keysets fail the hash-only and binary shapes; the bound is half the pool on both layouts, and ten opens in one join_all leave get_item and delete_table answering in milliseconds; a reader on a GSI, LSI or vector-index table takes DeleteTable down the asynchronous path with the row kept and every data table gone within a second of the release; a legacy-collation table of 100k rows streams unordered with a warning and no sort; a MongoDB document that does not decode ends the stream and a standalone server answers Unsupported; the plain-reader DELETING wait is 3 to 5 ms against 51.8 s before. The two COLLATE "C" mutants still pass, and the reason given holds: a C and a POSIX column sort alike on the server, so no ordered table reaches a clause that changes anything.

On the code added since: twenty concurrent DeleteTables on a ten-connection shared pool with readers on ten tables all returned within 0.52 s and left no orphan; an open holds one connection per pool on both layouts; twenty streaming snapshots did not move PutItem's p50 or p95 at 200 per second; the branch assertion fails for a slow UpdateTable and a flipped expectation; the release binary carries no test-hook symbol; both toolchains and the CI wiring are as the description says. Fourteen of fifteen mutants fail a test. Two inline items need code. On one shared database the single-transaction open holds no row lock before its LOCK TABLE, and a DeleteTable that probes in that gap commits the catalog removal and waits in DROP TABLE with no timeout: in 6 of 80 unaided open-and-delete races the data tables were present with no catalog row (0 of 40 on a separate data database). And the release wake is inert while a drop is deferred: the active loop sleeps on the timer alone, removal times with and without the wake are the same, both notify_one calls can be removed with all eight *_wakes_the_poller cases passing, and three comments and the description's "about 160 ms" describe a mechanism the code does not have. Four smaller items are inline: the one-second bound does not hold across a server restart; the poller never leaves its one-second loop after a drop failure that persists; the troubleshooting query misses a reader of an index table and states the wrong cadence; and the conformance check clears restore_summary on a false premise. The control_plane_delay_seconds row keeps a default of 5 where the schema inserts 0.25, and the new sentence applies at the shipped default; that text predates this revision and is not a condition.

Once the inline items are fixed, I will approve.

LeeroyHannigan · 2026-10-28T13:28:03Z

+            false
+        };
+
+        if delay_secs < 1.0 && !data_table_in_use {


On one shared database the open no longer holds the tables row, so a DeleteTable that probes between the open's first read and its LOCK TABLE takes this synchronous path, commits the catalog removal, then waits in DROP TABLE behind the stream with no lock timeout. I started an open and a DeleteTable of the same table together (the delete 0 to 4 ms later), held the stream 3 s and gave the caller a 2 s timeout; in 6 of 80 races the base and GSI data tables were present after the stream ended and a control-plane pass ran, with no catalog row, and on a separate data database 0 of 40 were. The catalog and the data are one database here, so run the drops in this transaction under a short lock_timeout before the removal commits, and take the asynchronous path when it times out.

LeeroyHannigan · 2026-10-28T13:28:03Z

+/// The transaction that holds the snapshot's `ACCESS SHARE`, from `BEGIN` to
+/// its end. Every way it ends wakes the control-plane poller: a
+/// `DeleteTable` that found the table in use left the row `DELETING`, and the
+/// wake lets the poller retry the drop at once rather than at its next


While a drop is deferred, the poller stays in its active loop, which sleeps in sleep_or_shutdown on the shutdown token and the one-second timer only, so it does not see this notify until it goes idle. With a snapshot held 12 to 14 s past DeleteTable and then dropped, the table was gone after 956, 40 and 13 ms on the separate layout and 911, 12 and 11 ms on the shared layout, the same spread as a plain reader that sends no wake, and removing the notify from release or from finish passes the eight *_wakes_the_poller cases. Either have the active loop wait on the notify as well as the timer, or correct this comment, the module doc at line 78, the in-catalog backup's comment and the description's "about 160 ms", which say the drop is retried at once.

LeeroyHannigan · 2026-10-28T13:28:03Z

+        // poller when they release; a plain transaction, pg_dump or a client
+        // read do not, so going idle while a drop is deferred would leave the
+        // row DELETING until the idle sweep (up to IDLE_TIMEOUT after the
+        // reader is gone). Staying active bounds that to about ACTIVE_POLL


After a restart this bound does not hold, because startup recovery runs one process_control_plane_transitions pass, which discards the deferred count, and the poller then starts in its idle wait. A 40 s reader that spanned a restart left the table DELETING for 37.2 s after its release, until the idle pass 60.0 s after the startup pass, on both layouts. Use control_plane_pass at startup and wake the poller when it reports a deferred drop.

LeeroyHannigan · 2026-10-28T13:28:03Z

                    "could not drop data tables for '{name}' ({table_id}); the table remains \
                     DELETING and the control plane will retry: {e}"
                );
+                deferred_drops += 1;


Every failed drop is counted here, so a drop that fails for a reason that persists keeps the poller in its one-second loop indefinitely. With a view on the data table, 120 s on the shared layout gave 0.97 failed drops per second, each logging this warning and each pass opening a restore-sweep connection, where before this change the poller went idle after its window and retried every 60 s. Count only lock timeouts (SQLSTATE 55P03) as deferred, or back off after repeated failures of another kind.

LeeroyHannigan · 2026-10-28T13:28:03Z

+/// 60 s idle pass; the wake makes the common case immediate (about 160 ms
+/// measured). `a_plain_reader_past_the_active_window_delays_the_drop_only_until_it_releases`
+/// pins the poller's own behaviour for a reader that sends no wake.
+async fn release_wakes_the_poller(shared: bool, how: Release) {


All eight cases of these tests pass with both notify_one calls removed from HeldSnapshot, and the removal times at the head match the plain reader's, which sends no wake (35 to 142 ms against 37 to 139 ms). The poller stays active while the drop is deferred, so the drop waiting in the lock queue is granted at the release whether or not the wake fires, and the in-catalog backup's wake has no test at all. Name the tests for what they check and drop the wake from the description's claims, or make the wake observable (the comment at snapshot_engine.rs line 155) and assert it.

LeeroyHannigan · 2026-10-28T13:28:03Z


 **Cause:** The background transition poller processes status changes when notified by CreateTable/DeleteTable, or on a 60-second defensive sweep. If the server was stopped while a table was in a transitional state, the transition completes on the next server startup.

+A second cause, on PostgreSQL only, is a session reading the table's data: a table snapshot, an in-catalog backup, a Scan or Query, `pg_dump`, or any client transaction holding `ACCESS SHARE` on one of the table's relations. DeleteTable then returns `DELETING` instead of dropping the tables in the request, and the poller retries the drop every second under a 3-second lock timeout until the reader is gone; the table disappears within about a second of the reader's release, and a CreateTable of the same name is refused with `ResourceInUseException` until then. A table that stays `DELETING` for longer has a long-running reader: find it with `SELECT pid, query_start, query FROM pg_stat_activity WHERE pid IN (SELECT pid FROM pg_locks WHERE mode = 'AccessShareLock' AND relation = to_regclass('"_ddb_<table_id>"'))` on the data database.


This query matches only the base data table, while DeleteTable defers on a reader of any of the table's relations; with a session holding only the GSI data table, DeleteTable returned DELETING and this query returned no rows on both layouts. Match every relation of the table (_ddb_<index_id> for each indexes row and _ddb_vec_<index_id> for each vector_indexes row) and add AND granted. The text says the poller retries every second; with a reader holding the table it tries about every 4 s (a 3 s lock wait, then a 1 s pause).

LeeroyHannigan · 2026-10-28T13:28:03Z

+/// The members of a description that are not fixed at the snapshot instant,
+/// cleared so two descriptions of one table compare equal: the table's and
+/// each index's item count and size are the stats pass's last values,
+/// `restore_summary` and `table_throughput_mode_summary` are set by the API


Storage sets restore_summary (build_table_description_on fills it on PostgreSQL and SQLite, and MongoDB reads it from the tables document), which is why the trait doc says it may be present on a restored table. Clearing it here lets a snapshot description that drops or misreports it compare equal. Compare it, and keep the clearing for the counts and table_throughput_mode_summary.

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), seven review rounds and the maintainer's reviews (on 2026-10-28 the sixth round, the maintainer's review and the seventh round) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction; the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and removes a FAILED restore's CREATING target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too). - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in its own transaction, and the ACTIVE flip sets every index ACTIVE. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8a adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; an http:// S3 endpoint needs force_path_style unless allow_insecure_endpoint is set; the credential clients' endpoints (STS, SSO, SSO OIDC and the general AWS_ENDPOINT_URL or profile endpoint_url) are held to the loopback rule, and a refusal names the client and the setting; S3 Express directory buckets are refused; runtime shutdown sends no abort for in-flight puts, so the server's shutdown awaits its jobs' failure paths) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. The endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. PR 7 adds put_if_absent to the store trait. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

…ackend Adds the one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code keeps its own copy of this logic for now; the engine-side backup job (next PR) is the first caller. Stacked on #394: SnapshotInfo is the manifest's SnapshotDescriptor (one definition, with the three KIND_* constants), and key_schema_is_orderable is KeyOrder::resolve, so a stream a backend marks ordered is one the manifest parser accepts and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. Trait (crates/storage/src/snapshot.rs): open_table_snapshot(account_id, table_name) returns TableSnapshot { description: TableDescription, info: SnapshotInfo, key_ordered: bool, items: BoxStream<Result<Item>> }. The description is the storage-level one at the snapshot instant, not the API response (restore_summary may be set, table_throughput_mode_summary is not, the counts are the stats pass's); items are yielded as stored (valid N normalized, invalid raw, TTL-expired unswept items included), and any error means start over, SnapshotExpired being the one worth retrying. The stream owns the backend transaction or session; dropping it releases it. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. The trait doc names what each backend holds while a stream is open and which errors each returns (Cassandra answers Unsupported before it resolves the table), the idle timeouts that end a stream as Internal, and why the trait has no default body and StorageError stays exhaustive; PostgreSQL and SQLite bound open snapshots per server (below) and refuse over the bound with LimitExceeded. StorageError gains SnapshotExpired for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient. Indexes: the description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since #401 UpdateTable creates the data table before committing the row, so this is a guard, not an expected state). A restore recreates the indexes in the description and builds them from the items, so a CREATING index with a data table comes back complete; one left out would not come back at all. PostgreSQL, one shared database: the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. Separate data database: the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key. A fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit); the byte budget is the backup job PR's. A table whose text key columns do not have a collation named "C" or "POSIX" (created before the key columns were declared with it; "default" on a C database also counts as not bytewise, which streams unordered and complete) is streamed unordered, with a warning: ordering it would turn the read into a sort of the whole table before the first row. SQLSTATE 40001 (recovery conflict on a standby) and 72000 (snapshot_too_old on a primary, PostgreSQL 16 and earlier) map to SnapshotExpired, pinned by a test that raises each code on the live server. The stream holds ACCESS SHARE on the data table until its last fetch, so a DROP TABLE waits for it; DeleteTable takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that). On both layouts the synchronous path takes every drop's lock before the catalog removal commits. It deletes the table's pending queue rows first, waiting for in-flight claims with no timeout, as #401 intends. It then drops the base, GSI, LSI and vector-index data tables, base first, within one 1 s budget for the group (SYNC_DROP_LOCK_TIMEOUT; before each DROP, lock_timeout is the time left). On the shared layout that runs in the catalog transaction behind a savepoint. With a separate data database it runs in a data transaction opened first, and a task the request awaits commits the catalog and then that transaction, so a cancelled caller cannot stop between the two. A lock wait that fails (55P03, a deadlock victim 40P01, or a statement_timeout's 57014) rolls the drops back and takes the asynchronous path with the row still held. A write transaction, a SHARE holder, an index build or a reader that starts after the probe therefore costs the request at most the 1 s, and no data table is left without its row. With a separate data database a data commit that fails after the catalog commit has the drop retried on a fresh connection for up to 60 s, so a failover, a dropped connection or a network blip no longer orphans the data tables; only a process that dies, or a data database unreachable for that whole window, between the catalog commit and the last retry leaves them without a row, and then they stay until an operator drops the relations the error log names (docs/troubleshooting.md, "Data tables without a catalog row"). The base table goes first because a write holds it before it writes an LSI or a synchronous GSI: with the index tables first, a write in progress waited about 1 s on the drop, and the drop went asynchronous. On the shared layout a DeleteTable probing between an open's snapshot and its LOCK does not see the open; if the LOCK is granted first, the drop times out after 1 s, is rolled back with the removal, and DeleteTable returns DELETING (in 80 unaided open-and-delete pairs, the delete first in half of them, 0 left a data table without a row on either layout; 39 to 40 deletes ran synchronously and 40 to 41 asynchronously). The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred, also after a restart: startup recovery runs the same pass and wakes the poller when it deferred a drop. A drop is deferred when the probe finds another session holding any lock on the table's relations, or when the drop fails in a lock wait (55P03, 40P01 or 57014); any other failure is logged and left to the normal window and the 60 s idle pass. Each pass probes a held table and skips it without waiting, and drops a free one within the same 1 s budget, so every held table is retried about once a second however many are held, and a CreateTable that is due is activated on the same cadence (1.1 s with four drops deferred). A drop that DeleteTable deferred is due at once, not after control_plane_delay_seconds. The active loop waits on the wake as well as on its timer, so a snapshot's or an in-catalog backup's release has the drop retried at once, and any holder's release is followed within about a second: 108 to 166 ms in the tests, with a 10 s test-only active poll for the wake cases. The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the default (0.25 on every backend), the bound and the retry cadence, the same-name CreateTable refusal until then, and a two-step query, over every relation of the table, that finds any session holding a lock on them and its lock mode. A DeleteTable of a table that is already DELETING returns its description and changes nothing. SQLite: one deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened and the WAL grows until the stream ends (a TRUNCATE checkpoint waits out the busy timeout and reports busy; the module doc gives the measured reply). #384's create_backup_from_wal_snapshot holds the writer lock while it pins because it also inserts the CREATING row; this path writes nothing. build_table_description_on reads the definition, including #384's restore_summary from table_restores, on that transaction. The stream holds a backup slot (backup_server_slots, shared with the in-catalog backup) until its read transaction ends (the short page that shows the end, or the first error) or it is dropped, so the pool arithmetic that keeps half the pool free counts snapshots; with no slot free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them. The pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index, so key_ordered is true for single-range-key tables; multi-part keys fall back to rowid order. A page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it has yielded its last item or is dropped, so the slots bound how many decoded copies exist, as #384's create_backup_single_connection does for the in-catalog backup. SQLite DeleteTable of a DELETING table (control_plane_delay_seconds of 1 or more) returns its description too. Cassandra: open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog BackupEngine path stays its backup path. A direct test asserts the refusal naming the backend on an existing table and a missing one. MongoDB: a session started with snapshot=true reads the tables document, the indexes documents (doc_to_table_description_in now takes an optional session) and the base collection at one cluster time; the marker is that time. Every read sets readConcern: snapshot explicitly. The production catalog handle carries readConcern: local (as a URI with readConcernLevel= would), and on it a read without the level fails: the first read reports no snapshot time, which the open refuses naming the read concern, and a later read in the pinned session is refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported; each of the two refusals names its own cause. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired, pinned by a unit test that builds the command error from a reply document; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes, CursorNotFound mapped to Internal). Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. key_ordered is false: for a composite key _id is a netstring of the key text, which is not the format's order for any key type. Test hooks (extenddb_storage::snapshot::paging): a page-size knob, a round-trip counter, and a pause at two points of the open (AfterDefinition, BeforeItems) that the conformance check holds while it commits writes. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with the test-hooks feature of extenddb-storage, which the test-hooks features of storage-postgres and storage-sqlite and the new snapshot-test-hooks of storage-mongodb enable (storage-mongodb's existing test-hooks, the integration binary's coordination hooks, is unchanged). The backend crates' tests enable them through a dev-dependency on themselves. Production builds carry neither the hooks nor the harness. PostgreSQL also gains test-hooks entries spawn_control_plane_poller_for_tests and spawn_control_plane_poller_with_active_poll_for_tests (the production poller, the second with its active retry interval as a parameter), recover_control_plane_at_startup_for_tests, control_plane_drop_attempts_for_tests (a per-engine count of drop attempts, a held table the probe skipped counted), arm_delete_table_pause_at (holds DeleteTable at BeforeDrops, between its probe and its drops, or at BetweenCommits, between the catalog commit and the data commit on the separate layout; arm_delete_table_pause is the BeforeDrops form), shorten_data_drop_retry_bound_for_tests (a process-global shorter bound for the retried data drop), and re-exports of the data relation names and of apply_all_migrations, so tests/table_snapshot.rs binds the crate's lists instead of copying them. The two pauses (the open's and DeleteTable's) are one generic type, extenddb_storage::test_pause::Pause, and snapshot_conformance::PageSize is the one page-size guard the three backends' tests use. tests/table_snapshot.rs fails instead of skipping without a connection string when EXTENDDB_TEST_PG_REQUIRED is set, which integration.yml sets. Tests: extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared page size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open (so a backend that ignored the knob, or a MongoDB cursor without batch_size, fails here instead of passing vacuously), then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. Asserts: 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, the counter advanced after the writer committed (a stream that read every page during the open fails), key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare (the format's comparator) when ordered, and a second snapshot taken after the writer stops shows the final round. Then, once per pause point, on a fresh table: with the open held, it commits W1, an UpdateTable adding a GSI, and W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the new GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts the snapshot's description must equal the storage layer's describe_table with the running counts and table_throughput_mode_summary cleared; restore_summary is compared, and each backend's suite also compares a table given a restore record (assert_snapshot_describes_the_table). Every drain is under a 120 s deadline naming the stage and the items read, and a test waiting for the hooks lock gives up when one holder has kept it 600 s, naming the wait. READ COMMITTED or no FOR SHARE on PostgreSQL, the data cursor or the index read outside the session on MongoDB, and the pauses moved outside the read window on SQLite or MongoDB, fail it, as do the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion), a description built from a second code path (the DescribeTable comparison), and a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3 with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. The suites pass on PostgreSQL 16 (both layouts), file-backed SQLite and MongoDB 7 (replica set; the standalone refusal against a standalone mongod), and the Cassandra test on Cassandra 4.1. PG: tests/table_snapshot.rs (81 tests and two ignored race measurements: consistency, marker and key order, tables with legacy-collation key columns streaming unordered and complete on a scratch database whose collation is checked not to be C, five held snapshots on a ten-connection pool leaving GetItem and DescribeTable unblocked with the sixth LimitExceeded and an open succeeding after one is dropped, a GSI without its data table left out with its attribute, DeleteTable during a snapshot returning DELETING, with the production poller at a 10 s active poll the table gone within 3 s of a release during the pause after the first drop attempt (for a dropped stream, a drained stream, a fetch error, an open cancelled at BeforeItems and the in-catalog backup), for a plain reader holding ACCESS SHARE that sends no wake, held 10 s past the poller's window, the table gone within 3 s of its release, a reader across a restart (the table gone within 3 s of its release), a drop failing on a dependent view making at most 3 attempts in the 15 s after the window, a restored table's restore_summary in the snapshot description, DeleteTable with only a GSI or vector-index data table locked taking the async path, an unparseable row ending the stream and freeing its permit, twenty concurrent DeleteTables on a ten-connection pool, an open completing with nine of ten connections held, an open held while nine DeleteTables queue, DeleteTable at BeforeItems taking the async path, a fetch error freeing the permit, a drained stream out of its transaction before its end is polled, a ROW EXCLUSIVE and a SHARE holder sending DeleteTable down the asynchronous path within about 1 s with no data table left without its row, a DeleteTable cancelled during that wait leaving the table ACTIVE with its data, write holders on the base and three GSI tables released 0.8 s apart taking it there at about 1 s, a write holding the base and then writing the GSI table finishing first with the table removed in the request, a claimed queue row holding DeleteTable until it ends, a probed reader sending it asynchronous in milliseconds, a statement timeout and a deadlock victim taking the asynchronous path, DeleteTable of a DELETING table returning its description and changing nothing, a deferred drop due at once, four held tables probed with no lock wait (the first released gone within 2 s, a CreateTable ACTIVE within 2 s), all on both pool layouts; on the separate layout, a data commit that fails after the catalog commit (its backend terminated between the two) retried until the drop goes through with the request returning DELETING and no relation left, a data database refusing connections past a shortened bound returning Internal naming every leaked relation with the row gone, and a caller cancelled between the two commits leaving no orphan; on the shared layout, a snapshot open or a GSI-only reader taking its lock after DeleteTable's probe sending it down the asynchronous path with the row and both data tables kept, and a table dropped before the shared-layout LOCK being TableNotFound; the SQLSTATE mapping; the two ignored race measurements, which delay the delete or the open in alternate pairs and assert both outcomes occurred) with a Drop guard that drops the scratch databases on success or panic, added to the direct-storage step in integration.yml; the full storage-postgres suite is 259 tests. SQLite: in-module tests (consistency, key order, the in-memory copy releasing the connection, in-memory copies holding their slot, a damaged attribute definition streaming unordered and complete, the rowid fallback at a page of seven with no duplicates and at least six fetches, slots shared with CreateBackup and its refusal text, the WAL checkpoint reply while a snapshot is held, a restored table's restore_summary compared on a file-backed and an in-memory engine, a drained stream and a stream that hit an error freeing their slot while still held, and an in-memory copy freeing its slot with its last item) with the same guard for the scratch files, created under the test binary's target directory. MongoDB: in-module tests (consistency, an undecodable document ending the stream, reads without the snapshot level failing with code 72 and the error text, the code mapping, a restored table's restore fields compared, the round-trip counter counting only the snapshot stream (a GSI backfill leaves it unchanged), and a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod container it starts (and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it)) on per-test scratch databases built without the bootstrapper, as #384's live tests do, the catalog handle with the production option set, dropped by a guard on success or panic, so a URI in the environment never makes cargo test touch another instance's databases; gated on EXTENDDB_TEST_MONGODB_URI (failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, which the CI job does), holding the snapshot hooks lock, run by a new integration-mongodb.yml job that starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod (both on 127.0.0.1 only). tests/test_table_operations.py: DeleteTable of a DELETING table returns its description, on PostgreSQL and SQLite (the state written into the catalog), skipped on MongoDB and Cassandra, which never leave a table DELETING. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), eight review rounds and the maintainer's reviews (on 2026-10-28 the sixth round, the maintainer's review and the seventh round; on 2026-10-29 the eighth, a sentence-level verification of the revision against the companions' code and main) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, and a data commit that fails after the catalog commit is retried on a fresh connection for a bounded time before the request returns (an orphan survives only a process death or a data database down past that bound, and the troubleshooting page says how to drop it); the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty, which never removes a target left for reason target_not_empty; that one is the operator's, removed with extenddb backup remove-target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle on the store's rows) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way, so one catalog owns a root at a time there too; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, remove-target, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too); main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows, and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8a adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; an http:// S3 endpoint needs force_path_style unless allow_insecure_endpoint is set; the credential clients' endpoints (STS, SSO, SSO OIDC and the general AWS_ENDPOINT_URL or profile endpoint_url) are held to the loopback rule, the IMDS endpoint setting may be http:// only to loopback or the two IMDS addresses, and a refusal names the client and the setting; S3 Express directory buckets and directory-bucket access points (--x-s3, --xa-s3) are refused; runtime shutdown sends no abort for in-flight puts, so the server's shutdown awaits its jobs' failure paths) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. The endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy, and the credential chain's clients get the same client when a credential endpoint setting, the IMDS endpoint setting or the container credentials URI is http://, so the chain's own http:// requests do not use the proxy; without such a setting the chain uses the SDK's default client and NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. PR 7 adds put_if_absent to the store trait. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

robinnsc requested review from LeeroyHannigan, amrith, c33howard, jcshepherd, pdf-amzn and yesyayen as code owners October 13, 2026 03:25

This was referenced Oct 13, 2026

fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384

Merged

docs(backup): design revision for off-host backup and restore [backup/restore 1/4] #396

Open

github-advanced-security Bot found potential problems Oct 13, 2026

View reviewed changes

Comment thread crates/storage-sqlite/src/backup.rs Fixed

This was referenced Oct 13, 2026

feat(backup-store): store trait with filesystem and S3 implementations [backup/restore 2/4] #393

Open

feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394

Open

robinnsc changed the title ~~feat(storage): SnapshotEngine, a consistent table snapshot on every backend~~ feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] Oct 13, 2026

robinnsc force-pushed the feat/table-snapshot branch from 3c2a30e to 12f5841 Compare October 13, 2026 06:50

robinnsc force-pushed the feat/table-snapshot branch from 12f5841 to 4e5a9b5 Compare October 26, 2026 02:38

robinnsc force-pushed the feat/table-snapshot branch from 4e5a9b5 to 8a38f20 Compare October 26, 2026 06:49

robinnsc changed the base branch from main to feat/backup-format-streaming October 26, 2026 06:49

robinnsc closed this Oct 26, 2026

robinnsc reopened this Oct 26, 2026

This was referenced Oct 26, 2026

fix(cassandra): a Query of a partition with no items left answers no items #611

Open

fix(cassandra): a Scan reads on past rows without an item #612

Open

robinnsc force-pushed the feat/backup-format-streaming branch from d157769 to bab52ff Compare October 26, 2026 20:02

robinnsc force-pushed the feat/table-snapshot branch from 8a38f20 to e6ec70a Compare October 26, 2026 20:02

LeeroyHannigan reviewed Oct 27, 2026

View reviewed changes

robinnsc force-pushed the feat/backup-format-streaming branch from bab52ff to 5500411 Compare October 27, 2026 20:50

robinnsc force-pushed the feat/table-snapshot branch from e6ec70a to c9b4a17 Compare October 27, 2026 20:50

robinnsc force-pushed the feat/backup-format-streaming branch from 5500411 to 4d731aa Compare October 28, 2026 00:02

robinnsc force-pushed the feat/table-snapshot branch from c9b4a17 to 0507485 Compare October 28, 2026 00:02

This was referenced Oct 28, 2026

Off-host backup and restore (tracking) #624

Open

Backup/restore: SnapshotEngine on every backend #628

Open

robinnsc force-pushed the feat/backup-format-streaming branch from 4d731aa to 6767bb7 Compare October 28, 2026 06:31

robinnsc force-pushed the feat/table-snapshot branch from 0507485 to 61e98ec Compare October 28, 2026 06:31

robinnsc self-assigned this Oct 28, 2026

LeeroyHannigan reviewed Oct 28, 2026

View reviewed changes

LeeroyHannigan added the v1 Needed for the 1.0 release label Oct 28, 2026

robinnsc force-pushed the feat/backup-format-streaming branch from 6767bb7 to 995af82 Compare October 29, 2026 02:47

robinnsc force-pushed the feat/table-snapshot branch from 61e98ec to dfbbe82 Compare October 29, 2026 02:48

robinnsc force-pushed the feat/backup-format-streaming branch from 995af82 to 3f7107d Compare October 29, 2026 08:11

robinnsc force-pushed the feat/table-snapshot branch from dfbbe82 to 069c4b1 Compare October 29, 2026 08:11


		Cause: The background transition poller processes status changes when notified by CreateTable/DeleteTable, or on a 60-second defensive sweep. If the server was stopped while a table was in a transitional state, the transition completes on the next server startup.

		A second cause, on PostgreSQL only, is a session reading the table's data: a table snapshot, an in-catalog backup, a Scan or Query, `pg_dump`, or any client transaction holding `ACCESS SHARE` on one of the table's relations. DeleteTable then returns `DELETING` instead of dropping the tables in the request, and the poller retries the drop every second under a 3-second lock timeout until the reader is gone; the table disappears within about a second of the reader's release, and a CreateTable of the same name is refused with `ResourceInUseException` until then. A table that stays `DELETING` for longer has a long-running reader: find it with `SELECT pid, query_start, query FROM pg_stat_activity WHERE pid IN (SELECT pid FROM pg_locks WHERE mode = 'AccessShareLock' AND relation = to_regclass('"_ddb_<table_id>"'))` on the data database.

Conversation

robinnsc commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing done

Not changed here

Checklist

Breaking changes

Uh oh!

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

robinnsc commented Oct 13, 2026 •

edited

Loading